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The Road to Redemption: 

Reclaiming the Value in Assessment Retention Exams 


Trey Stohlman 1 

Abstract: A good assessment plan combines many direct and indirect measures to 
validate the collected data. One often controversial assessment measure comes in 
the form of retention exams. Although assessment retention exams may come with 
faults, others advocate for their inclusion in program assessment. Objective-based 
tests may offer insight to instructors about course objectives that students 
comprehend well and other concepts that need more attention. This research shows 
that using retention exams as an assessment measure can generate useful and 
meaningful data for both the students and the program. Students can learn 
strengths and weaknesses based on scores. Faculty and programs can learn where 
gaps may exist within the program. But, whenever a program decides to use 
retention exams as an assessment tool, faculty members need to be included in the 
process. Discussions about content need to occur constantly. Exams need to 
consistently reflect current standards and student learning objectives. And faculty 
need to stay involved in the process to know exactly where any inconsistencies may 
lie within their courses, and how they contribute to the students ’ overall experience 
within the program. 
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Many instruments exist for program assessment. Some use direct measures such as final course 
grades which show comprehension at the completion of the course. Others use indirect measures 
like feedback from alumni. These data from alumni give organizations feedback about what is 
currently happening in the industry to ensure students stay competitive with their peers at other 
institutions. A good assessment plan combines many direct and indirect measures to validate the 
collected data. “Because any one assessment is imperfect and imprecise, collect more than one 
kind of evidence of what students have learned” (Suskie, 2009, p. 38). One often controversial 
assessment measure comes in the form of retention, or exit exams. Banta and Palomba (2015) 
defined these objective-based exams as instruments that allow students to “demonstrate the 
knowledge they have acquired and their ability to process and use that knowledge” (p. 105). In 
comparing these tests to those used in elementary and secondary school. Tucker (2006) defined 
exit exams as “tests that cover specific material deemed by state or federal officials as important 
for students at that particular level” (p. 374). The exams in this case combine these two definitions, 
resulting in tests that measure student retention of knowledge of core course concepts in a given 
subject. 

Tucker (2006) identified one criticism of retention exams as the material covered on the 
exam often dates back to the beginning of a student’s collegiate experience. The department will 
find itself testing students on material taken several years earlier. If this material has been 
reinforced as the student has gone through the program, this will not be a problem. If, however, it 
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has been 5 or 6 years since the student has been exposed to the material, then review sessions 
might be required (Tucker, 2006, p. 379-80). 

Although assessment retention exams may come with faults, others advocate for their 
inclusion in program assessment. Banta and Palomba (2015) argued that “tests that contain well- 
written items covering the appropriate subject matter and level of thinking can reveal much about 
student learning” (p. 110). Objective-based tests may offer insight to instructors about course 
objectives that students comprehend well and other concepts that need more attention. Tucker 
(2006) agreed and said to make the exams become part of a regularly graded class such as a 
capstone experience. Students expect these types of exams to be part of a course because they 
expect to be tested on course material. Since the capstone class encompasses the entire program, 
students expect some sort of testing of their knowledge and skills of the program prior to their 
graduation. In the case of the broadcasting program at the university used as the subject for this 
paper, the exams became part of the capstone class, and featured appropriate subject matter to 
reveal true evidence of student learning. However, the road to recovery for this assessment measure 
hit a few bumps along the way. This paper shares the importance and the value of a systematic 
approach to assessment regarding the data gathered, and also provides a way to create commitment 
within the entire department or program to strengthen their inclusion as an effective assessment 
tool. 

Initial Analysis of Retention Exams 

When first presented with the capstone course and its contents, the instructor, as any new instructor 
to a course might do, utilized the resources from the previous instructor. Although the materials 
were comprehensive, the assessment exam portion of the class lacked efficiency in the execution 
of the exams and consistency of the construction of the exams as well. The first issue the instructor 
addressed was the efficiency in the distribution and completion of the exams. The instructor 
utilized the learning management system in use by the university to construct electronic 
examinations so tests could be immediately scored, and so data from the tests could be collected 
and aggregated more efficiently. This only scratched the surface of the issue. The collected data 
still reported inaccurate information as the tests were not normed or consistent. Tucker (2006) 
found that “given the lack of nationally normed exams in the area of mass communication, any 
department that decides to use an exit exam as a measure of direct assessment will have to create 
one” (p. 378). The broadcast department in this study constructed the exams, but lacked the 
appropriate guidance to make them able to collect rich, useful assessment data. The second step to 
redemption for the retention exams as an assessment instrument needed to tackle this issue. 

A New Paradigm 

The key to creating useful retention exams for assessment starts with involving the faculty in their 
creation. Banta and Palomba (2015) agreed that “tests that are developed locally for program 
assessment typically reflect a group effort” (p. 108). Suskie (2009) also pointed out that “our 
assignments are more effective if we first clarify what we want students to leam from the 
assignment and then design an assignment that will help them achieve those ends” (p. 148) and 
objective tests “are especially good for assessing fundamental knowledge and understanding” (p. 
166). As a result, the instructor charged the faculty with creating retention exams that reflected 


Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 5, October, 2015. 
Josotl.lndiana.edu 


65 



Stohlman 


fundamental knowledge and understanding, and emphasized retention of course and program 
objectives. 

The retention exam creation started with a test blueprint. A good objective test that uses a 
test blueprint ensures that tests reflect course objectives and learning goals deemed important by 
the department (Banta & Palomba, 2015; Suskie, 2009). The department simplified this process 
by using the Master Course Syllabus (MCS) required for every course as the template. The MCS, 
required by the university, outlines each course by providing a description, Student Learning 
Outcomes/Objectives (SLO), and suggested course evaluation measures. The student learning 
outcomes and course objectives listed on the MCS clearly indicate the desired outcomes of the 
course. Each question on the exam needed to reflect these measures and not any text- or instructor- 
specific content. Although the MCS simplified the process, the test creators still needed some 
additional guidance. Tucker (2006) suggested four helpful techniques: 

1. The content of the tests should match the classroom objectives and what the teacher 
emphasized. 

2. The test items should represent the full range of knowledge and skills that are the primary 
targets of instruction. 

3. Expectations for student performance should be clear. 

4. The assessment should be free of extraneous factors, which unnecessarily confuse or 
inadvertently cue student responses, (p. 375) 

The broadcast department faculty created the new retention examinations under the 
guidance of the assessment coordinator and course instructor. Each instructor reviewed the existing 
version of the exam to compare questions on the exam against student learning outcomes on the 
MCS. Any question that did not reflect one of the course objectives was eliminated and replaced 
with one that did. For example, one SLO required students to “identify a variety of methods of 
mass media criticism and analysis including aesthetic, sociological, economic, structural, 
psychoanalytical and ethical perspectives.” The original multiple choice question asked students 
to define “the Way of the Rhetorician.” This was a question specific to the textbook, and not 
covered by all instructors of the course. A new question addressing this SLO asked students, in 
multiple choice format, to define Auteur Theory. This new question addressed a basic tenet of the 
course, covered by all instructors of the course. Each instructor created their objective-based exams 
in this manner with multiple choice, true/false, and matching-style questions using the MCS 
objectives as a blueprint. 

The following semester the capstone course instructor deployed the new exams through 
the learning management system as before. Although the new tests generated improvements in 
scores, new concerns arose from the faculty, and new problems occurred in the classroom. 

The faculty questioned, now that the exams were taken electronically though the learning 
management system, if the capstone course was the appropriate place to administer the exams. 
Students previously enrolled in the course were not held accountable for their performance on the 
exams, so the faculty questioned the authenticity of the data and the purpose of their inclusion in 
the course. The instructor and assessment coordinator argued for the continued inclusion of the 
exams as part of the capstone class for the program. Suskie (2009) supported this argument, stating 
“capstones help students synthesize their learning by tying together the various elements of their 
program and seeing the big picture” (p. 7). The assessment retention exams administered in the 
course reflected this synthesis. Additionally, Banta and Palomba (2015) agreed that “objective 
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tests are a normal and expected part of the classroom experience and are a type of direct measure 
included in many assessment programs” and “objective tests allow students to demonstrate the 
knowledge they have acquired and their ability to process and use that knowledge” (p. 105). So 
the exams stayed. But the question then became not about whether the exams should stay part of 
the capstone experience, but how to hold the students accountable for their performance. 

Addressing Accountability 

The assessment coordinator for the program suggested including the exam scores as part of the 
course grade. Banta and Palomba (2015) supported this argument noting “with locally developed 
assessment instruments, faculty are often more comfortable with test content and its relationship 
to the curriculum, and are more likely to include results in the course grade” (p. 109). Faculty 
questioned the fairness of including scores on test material covered in courses anywhere from two 
to five years prior to when the students take the exams. Informing the students prior to taking the 
exams of their value, and the inclusion in their course grade effectively tackles this issue. Banta 
and Palomba (2015) addressed this concern as well, noting that “because of the many possible 
approaches, students must be informed of the effect, if any, that their performance or participation 
will have on their grades” (p. 109). The course instructor suggested holding the students 
accountable for only a portion of the scores on the exams. This way the exams would encourage 
the students to perform at their highest level without penalizing them too severely for poor 
retention. The faculty seemed amenable to the compromise. But, this required more changes to the 
exams. 

Each exam needed to contain the same number of questions to simplify the point process. 
Students enrolled in the capstone class completed a total of eight different assessment exams, one 
for each core class in the curriculum and one elective exam reflecting their chosen concentration 
within the program. The proposed method for inclusion scored all eight exams, but only held the 
students accountable for their performance on seven. The instructor aggregated the scores on the 
exams and divided by one less. Students who performed well could potentially score higher than 
100%, but students who failed to perform well would not suffer harshly for their performance. 
With all parties in agreement and new tests to deploy, the new semester looked promising for the 
assessment coordinator and capstone course instructor. 
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Data Collection and Analysis 

Data from the assessment exams in the semester following deployment of the new procedure 
provided encouraging results. 

Table 1. Comparison results from old instrument to new instrument 



Course 

Year 

Results 

Goal Changes from 2013 to 2014 


1 . 

Survey 

12-13 

19.8% earned 70% or greater 

80% of achieving 75% or better 




13-14 

53.1% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 

2. 

Audio 

12-13 

26.4% earned 75% or greater 

80% of achieving 75% or better 




13-14 

36.3% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 

3. 

Video 

12-13 

54.4% earned 75% or greater 

80% of achieving 75% or better 




13-14 

38.1% earned 75% or greater 

new goal 75% achieving 75% 
better 

or 

4. 

New Tech 

12-13 

79.7% earned 75% or greater 

80% of achieving 75% or better 




13-14 

67.5% earned 75% or greater 

new goal 75% achieving 75% 
better 

or 

5. 

Copywriting 

12-13 

68.4% earned 75% or greater 

80% of achieving 75% or better 




13-14 

74.5% earned 75% or greater 

new goal 75% achieving 75% 
better 

or 

6. 

News writing 

12-13 

25.0% earned 75% or greater 

80% of achieving 75% or better 




13-14 

66.7% earned 75% or greater 

new goal 75% achieving 75% 
better 

or 

7. 

Web Content 

12-13 

38.1% earned 75% or greater 

80% of achieving 75% or better 




13-14 

51.5% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 

8. 

Criticism 

12-13 

31.4% earned 75% or greater 

80% of achieving 75% or better 




13-14 

40.0% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 

9. 

Law & Policy 

12-13 

4.5% earned 75% or greater 

80% of achieving 75% or better 




13-14 

38.0% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 


Overall 

12-13 

18.7% earned 75% or greater 

80% of achieving 75% or better 




13-14 

48.0% earned 70% or greater 

new goal 70% achieving 70% 
better 

or 


Students scored higher than in previous semesters thanks to uniform exams that reflected course 
objectives. Banta and Palomba (2015) mentioned that incentives for students to participate in 
assessment activities may be necessary, but noted that intrinsic motivation always elicits their best 
work. Tying the score on the exams to their course grade worked as the motivating factor in this 
case. Additionally, the instructor communicated the importance of the exams to the students, 
noting that the data collected help to make the courses in the program stronger for future students. 
“The messages faculty give about assessment are powerful motivators. If faculty care about 


Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 5, October, 2015. 
Josotl.lndiana.edu 


68 



Stohlman 


assessment, students are much more likely to care too” (Banta & Palomba, 2015, p. 62). The 
inclusion of the scores in the final course grade, and the discussion of the implications of the 
results, encouraged students to perform at their best level. 

The electronic distribution of the exams allowed the department to collect data for 
instructors to evaluate which concepts needed more attention, and those concepts students retained 
more successfully. 

Table 2. Sample Question 

Answers Question 17: Auteur Theory % answered 

A is limited to European program content 0% 

B was developed by Herbert Gans 5.882% 

C is applicable when the creator puts a recognizable stamp on the work 0% 

D is a key form of originator criticism 5.882% 

E both c and d (Correct Answer) 88.235% 

The instructors used this data to ensure instruction of course concepts remained even across 
all instructors, and that those concepts where students were lacking in retention were reinforced. 
Additionally, instructors could alter exams if questions were continually missed. Although the 
process still needed some adjusting, the introductory phase of deployment encouraged the course 
instructor and assessment coordinator to continue their efforts. 


Determine 

Distribution 


Clarify SLO. 
Develop a 
blueprint 


Revise Root questions 

Refine nl SLO 


Determine 

Accountability 


Figure 1: Development Cycle 
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Conclusions 

The use of assessment retention exams is back on the rise. Banta and Palomba (2015) noted that 
“the percentage of program heads reporting that they use this assessment method is above 50 
percent for trade programs, health sciences, computer science, business, and engineering” (p. 106). 
In the broadcasting program at the university in this paper, assessment efforts continue to look for 
ways to use multiple assessment instruments to gather useful, meaningful, and rich data. Suskie 
(2009) reminded programs and assessment coordinators to “make participation in the assessment 
a requirement of a program or course (typically a capstone course)” (p. 29). Banta and Palomba 
(2015) prompted test designers to ensure that tests are not focused at “such a general level of 
information that they do not yield detailed results useful for improvement of teaching and learning” 
(p. 107). 

This research shows that using retention exams as an assessment measure can generate 
useful and meaningful data for both the students and the program. Students can learn strengths and 
weaknesses based on scores. Faculty and programs can leam where gaps may exist within the 
program. But, whenever a program decides to use retention exams as an assessment tool, faculty 
members need to be included in the process. Discussions about content need to occur constantly. 
Exams need to consistently reflect current standards and student learning objectives. And faculty 
need to stay involved in the process to know exactly where any inconsistencies may lie within their 
courses, and how they contribute to the students’ overall experience within the program. 

The data continue to flow, and the results continue to generate useful information for 
program evaluation. But the process needs to continue to constantly be reevaluated. Without 
continuous monitoring and attention to exams to ensure their reflection of changing course 
objectives, the critics of retention exams will continue to devalue retention exams as useful 
assessment measures. For this program, the road to redemption lead to richer more useful data than 
any collected before. 
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